{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "hw2.ipynb.txt",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S2IaITBP0f5R"
      },
      "source": [
        "# Homework 2\n",
        "In this homework, we will explore language generation using character-level RNNs. Sounds awesome, right???\n",
        "\n",
        "A few notes at the beginning:\n",
        "- It might be useful for you to read the whole assignment before beginning. Especially the last two sections so you know what to record for turning in.\n",
        "- Much of the required knowledge in this (and past) homeworks about Python, PyTorch, etc. are not explained fully here. Instead, we expect you to use the existing documentation, search engines, Stack Overflow, etc. for implementation details.\n",
        "- That being said, we have listed several functions in parts of the homework where knowing those functions exist would be especially useful. However you will still need to read the docs on how to specifically use the functions.\n",
        "\n",
        "# Part 0: Initial setup\n",
        "You should recognize this code from last time.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4GS0yuGl0mHQ"
      },
      "source": [
        "import torch\n",
        "print('Version', torch.__version__)\n",
        "print('CUDA enabled:', torch.cuda.is_available())\n",
        "  \n",
        "# Running this should then print out:\n",
        "# Version 1.7.0+cu101 (or something like this)\n",
        "# CUDA enabled: True"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6t3ZIEll0pr-"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/gdrive')\n",
        "!ls /gdrive"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PEzPNAIY0vkm"
      },
      "source": [
        "# Part 1: Upload the dataset\n",
        "We will be using the complete text of Harry Potter as our corpus. We will provide it for you in a not-very-well-formatted way.\n",
        "Run this code to set up the homework3 files.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CLVJPc_90vsB"
      },
      "source": [
        "import os\n",
        "BASE_PATH = '/gdrive/My Drive/colab_files/hw2/'\n",
        "if not os.path.exists(BASE_PATH):\n",
        "    os.makedirs(BASE_PATH)\n",
        "DATA_PATH = '/content/'\n",
        "\n",
        "if not os.path.exists(os.path.join(DATA_PATH, 'harry_potter.txt')):\n",
        "    os.chdir(BASE_PATH)\n",
        "    !wget https://courses.cs.washington.edu/courses/cse599g1/19au/files/homework3.tar.gz\n",
        "    !tar -zxvf homework3.tar.gz\n",
        "    !rm homework3.tar.gz\n",
        "    !cp pt_util.py /content\n",
        "    !cp harry_potter.txt /content\n",
        "os.chdir('/content')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hd1Qx66s19Pl"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "from torchvision import datasets\n",
        "from torchvision import transforms\n",
        "import numpy as np\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "import sys\n",
        "import pickle\n",
        "import re\n",
        "import pt_util"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AxIvm7h62tfx"
      },
      "source": [
        "#Part 2: Preprocessing the data\n",
        "In previous homeworks, we have provided a cleaned version of the data. But this time you'll have to do some of that cleaning yourselves.\n",
        "\n",
        "Hints:\n",
        "- train_text and test_text should contain the class indices for the character tokens from the data file. For example, if the text was **`\"ABA CDBE\"`**, the token version would be a numpy array with contents `[0, 1, 0, 2, 3, 4, 1, 5]`\n",
        "- The harry_potter.txt file has weird spacing. You might want to replace all the whitespace characters (space, \\n, \\t, etc.) in the file with the space character.\n",
        "- You should output two files. One for training and one for testing. The training should be the first 80% of the characters.\n",
        "- voc2ind is a map from character to the index of the class for that character. There is no predefined vocabulary, but you will need to be consistent across all tasks that use the vocabulary. For the example above, the voc2ind would be `{'A': 0, 'B': 1, ' ': 2, 'C': 3, 'D': 4, 'E': 5}`\n",
        "- ind2voc is the inverse of voc2ind\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6oZq_S6k3GpB"
      },
      "source": [
        "def prepare_data(data_path):\n",
        "    with open(data_path) as f:\n",
        "        # This reads all the data from the file, but does not do any processing on it.\n",
        "        data = f.read()\n",
        "    \n",
        "    # TODO Add more preprocessing\n",
        "    \n",
        "    voc2ind = {}\n",
        "    \n",
        "    # Compute voc2ind and transform the data into an integer representation of the tokens.\n",
        "    for char in data:\n",
        "        pass # TODO Fill this in\n",
        "\n",
        "\n",
        "    ind2voc = {val: key for key, val in voc2ind.items()}\n",
        "\n",
        "    train_text = None # TODO Fill this in\n",
        "    test_text = None # TODO Fill this in\n",
        "\n",
        "    pickle.dump({'tokens': train_text, 'ind2voc': ind2voc, 'voc2ind':voc2ind}, open(DATA_PATH + 'harry_potter_chars_train.pkl', 'wb'))\n",
        "    pickle.dump({'tokens': test_text, 'ind2voc': ind2voc, 'voc2ind':voc2ind}, open(DATA_PATH + 'harry_potter_chars_test.pkl', 'wb'))\n",
        "    \n",
        "prepare_data(DATA_PATH + 'harry_potter.txt')\n",
        "\n",
        "\n",
        "class Vocabulary(object):\n",
        "    def __init__(self, data_file):\n",
        "        with open(data_file, 'rb') as data_file:\n",
        "            dataset = pickle.load(data_file)\n",
        "        self.ind2voc = dataset['ind2voc']\n",
        "        self.voc2ind = dataset['voc2ind']\n",
        "\n",
        "    # Returns a string representation of the tokens.\n",
        "    def array_to_words(self, arr):\n",
        "        return ''.join([self.ind2voc[int(ind)] for ind in arr])\n",
        "\n",
        "    # Returns a torch tensor representing each token in words.\n",
        "    def words_to_array(self, words):\n",
        "        return torch.LongTensor([self.voc2ind[word] for word in words])\n",
        "\n",
        "    # Returns the size of the vocabulary.\n",
        "    def __len__(self):\n",
        "        return len(self.voc2ind)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kzX1tUv8ilYV"
      },
      "source": [
        "#Part 3: Loading the data\n",
        "This is possibly the trickiest part of this homework. In the past, batches were not correlated with each other, and the data within a single minibatch was also not correlated, so you could basically draw randomly from the dataset. That is not the case here. Instead, you should return sequences from the dataset.\n",
        "\n",
        "Your instructions are to implement the following. First, imagine splitting the dataset into N chunks where N is the batch_size and the chunks are contiguous parts of the data. For each batch, you should return one sequence from each of the chunks. The batches should also be sequential an example is described below.\n",
        "\n",
        "The data is 20 characters long `[1, 2, 3, ...20]`. The batch size is `2` and the sequence length is `4`\n",
        "- The 1st batch should consist of  `(data =  [[1, 2, 3, 4]; [11, 12, 13, 14]], labels = [[2, 3, 4, 5]; [12, 13, 14, 15]])`\n",
        "- The 2nd batch should consist of `(data =  [[5, 6, 7, 8]; [15, 16, 17, 18]], labels = [[6, 7, 8, 9]; [16, 17, 18, 19]])`\n",
        "- The 3rd batch should consist of `(data =  [[9]; [19]], labels = [[10]; [20]])`\n",
        "- There is no 4th batch.\n",
        "\n",
        "Hints:\n",
        "- To work with the rest of the code, your len(dataset) should be a multiple of the batch_size. \n",
        "- Removing the last bit to make the data the proper shape will probably give better results than padding with 0s.\n",
        "- It is OK to have one batch be shorter than the others as long as all entries in that batch are the same length.\n",
        "- Notice that the last label in one batch is the first data in the next batch. Be careful of off-by-one errors.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "44v6o0JwiwXk"
      },
      "source": [
        "class HarryPotterDataset(torch.utils.data.Dataset):\n",
        "    def __init__(self, data_file, sequence_length, batch_size):\n",
        "        super(HarryPotterDataset, self).__init__()\n",
        "\n",
        "        self.sequence_length = sequence_length\n",
        "        self.batch_size = batch_size\n",
        "        self.vocab = Vocabulary(data_file)\n",
        "\n",
        "        with open(data_file, 'rb') as data_pkl:\n",
        "            dataset = pickle.load(data_pkl)\n",
        "\n",
        "        # TODO: Any preprocessing on the data to get it to the right shape.\n",
        "\n",
        "\n",
        "    def __len__(self):\n",
        "        # TODO return the number of unique sequences you have, not the number of characters.\n",
        "         raise NotImplementedError \n",
        "        \n",
        "    def __getitem__(self, idx):\n",
        "        # Return the data and label for a character sequence as described above.\n",
        "        # The data and labels should be torch long tensors.\n",
        "        # You should return a single entry for the batch using the idx to decide which chunk you are \n",
        "        # in and how far down in the chunk you are.\n",
        "        \n",
        "        # TODO\n",
        "        data = None\n",
        "        return data[:-1], data[1:]\n",
        "\n",
        "    def vocab_size(self):\n",
        "        return len(self.vocab)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8kYKDZoj2jCV"
      },
      "source": [
        "# Part 4: Defining the Network\n",
        "This time we will provide a network that should already get pretty good performance. You will still need to write the forward pass and inference functions. You may also choose to modify the network to try and get better performance.\n",
        "\n",
        "__BE CAREFUL:__ We have specified that the data will be fed in as batch_first. Look at the documentation if you are confused about the implications of this as well as how to call it for the forward pass. https://pytorch.org/docs/stable/nn.html#torch.nn.GRU\n",
        "    \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mO21UXLj2ixn"
      },
      "source": [
        "TEMPERATURE = 0.5\n",
        "\n",
        "class HarryPotterNet(nn.Module):\n",
        "    def __init__(self, vocab_size, feature_size):\n",
        "        super(HarryPotterNet, self).__init__()\n",
        "        self.vocab_size = vocab_size\n",
        "        self.feature_size = feature_size\n",
        "        self.encoder = nn.Embedding(self.vocab_size, self.feature_size)\n",
        "        self.gru = nn.GRU(self.feature_size, self.feature_size, batch_first=True)\n",
        "        self.decoder = nn.Linear(self.feature_size, self.vocab_size)\n",
        "        \n",
        "        # This shares the encoder and decoder weights as described in lecture.\n",
        "        self.decoder.weight = self.encoder.weight\n",
        "        self.decoder.bias.data.zero_()\n",
        "        \n",
        "        self.best_accuracy = -1\n",
        "    \n",
        "    def forward(self, x, hidden_state=None):\n",
        "        batch_size = x.shape[0]\n",
        "        sequence_length = x.shape[1]\n",
        "        \n",
        "        # TODO finish defining the forward pass.\n",
        "        # You should return the output from the decoder as well as the hidden state given by the gru.\n",
        "        raise NotImplementedError \n",
        "\n",
        "        return x, hidden_state\n",
        "\n",
        "    # This defines the function that gives a probability distribution and implements the temperature computation.\n",
        "    def inference(self, x, hidden_state=None, temperature=1):\n",
        "        x = x.view(-1, 1)\n",
        "        x, hidden_state = self.forward(x, hidden_state)\n",
        "        x = x.view(1, -1)\n",
        "        x = x / max(temperature, 1e-20)\n",
        "        x = F.softmax(x, dim=1)\n",
        "        return x, hidden_state\n",
        "\n",
        "    # Predefined loss function\n",
        "    def loss(self, prediction, label, reduction='mean'):\n",
        "        loss_val = F.cross_entropy(prediction.view(-1, self.vocab_size), label.view(-1), reduction=reduction)\n",
        "        return loss_val\n",
        "\n",
        "    # Saves the current model\n",
        "    def save_model(self, file_path, num_to_keep=1):\n",
        "        pt_util.save(self, file_path, num_to_keep)\n",
        "\n",
        "    # Saves the best model so far\n",
        "    def save_best_model(self, accuracy, file_path, num_to_keep=1):\n",
        "        if accuracy > self.best_accuracy:\n",
        "            self.save_model(file_path, num_to_keep)\n",
        "            self.best_accuracy = accuracy\n",
        "\n",
        "    def load_model(self, file_path):\n",
        "        pt_util.restore(self, file_path)\n",
        "\n",
        "    def load_last_model(self, dir_path):\n",
        "        return pt_util.restore_latest(self, dir_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEQZIoB0jY5h"
      },
      "source": [
        "#Part 5: Character Generation\n",
        "\n",
        "In class we discussed three algorithms for creating sequences.\n",
        "1. Max: Choose the most likely value\n",
        "2. Sample: Sample from the distribution output by the network.\n",
        "3. Beam Search: Sample from the distribution and use the Beam Search algorithm.\n",
        "\n",
        "The beam search algorithm is as follows:\n",
        "```\n",
        "1. Initialize the beam list with the single existing empty beam\n",
        "2. Repeat for the sequence length:\n",
        "    1. For each beam in the beam list:\n",
        "        1. Compute the next distribution over the output space for that state\n",
        "        2. Sample from the distribution with replacement\n",
        "        3. For each sample:\n",
        "            1. Compute its score\n",
        "            2. Record its hidden state and chosen value\n",
        "        4. Add all the samples to the new beam list      \n",
        "     2. Rank the new beam list\n",
        "     3. Throw out all but the top N beams\n",
        " 3. Return the top beam's chosen values.\n",
        "```\n",
        "\n",
        "\n",
        "Hints:\n",
        "- np.random.choice and torch.multinomial will both help with the sampling as they can take in a weighted probability distribution and sample from that distribution.\n",
        "- For beam search you will need to keep a running score of the likelihood of each sequence. If you multiply the likelihoods, you will encounter float underflow. Instead, you should add the log likelihoods.\n",
        "- For beam search, you will need to keep track of multiple hidden states related to which branch you are currently expanding.\n",
        "- For beam search, you should search over the beam, but only return the top result in the end.\n",
        "- It may be useful to do the training part before the character generation part so you have some model to test.\n",
        "- Feel free to play around with the `BEAM_WIDTH`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9XTxy4eq3UYR"
      },
      "source": [
        "TEMPERATURE = 0.5\n",
        "BEAM_WIDTH = 10\n",
        "\n",
        "def max_sampling_strategy(sequence_length, model, output, hidden, vocab):\n",
        "    outputs = []\n",
        "    for ii in range(sequence_length):\n",
        "        # TODO\n",
        "        raise NotImplementedError\n",
        "    return outputs\n",
        "    \n",
        "def sample_sampling_strategy(sequence_length, model, output, hidden, vocab):\n",
        "    outputs = []\n",
        "    for ii in range(sequence_length):\n",
        "        # TODO\n",
        "        raise NotImplementedError\n",
        "    return outputs\n",
        "\n",
        "def beam_sampling_strategy(sequence_length, beam_width, model, output, hidden, vocab):\n",
        "    outputs = []\n",
        "    beam = [([], output, hidden, 0)]\n",
        "    # TODO\n",
        "    raise NotImplementedError\n",
        "    return outputs\n",
        "\n",
        "\n",
        "def generate_language(model, device, seed_words, sequence_length, vocab, sampling_strategy='max', beam_width=BEAM_WIDTH):\n",
        "    model.eval()\n",
        "\n",
        "    with torch.no_grad():\n",
        "        seed_words_arr = vocab.words_to_array(seed_words)\n",
        "\n",
        "        # Computes the initial hidden state from the prompt (seed words).\n",
        "        hidden = None\n",
        "        for ind in seed_words_arr:\n",
        "            data = ind.to(device)\n",
        "            output, hidden = model.inference(data, hidden)\n",
        "        \n",
        "        if sampling_strategy == 'max':\n",
        "            outputs = max_sampling_strategy(sequence_length, model, output, hidden, vocab)\n",
        "\n",
        "        elif sampling_strategy == 'sample':\n",
        "            outputs = sample_sampling_strategy(sequence_length, model, output, hidden, vocab)\n",
        "\n",
        "        elif sampling_strategy == 'beam':\n",
        "            outputs = beam_sampling_strategy(sequence_length, beam_width, model, output, hidden, vocab)\n",
        "\n",
        "\n",
        "        return vocab.array_to_words(seed_words_arr.tolist() + outputs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Havsk_RJi_i5"
      },
      "source": [
        "# Part 6: Training\n",
        "Again, we are providing training code for you. Have a look at the train function though as it implements the exact forward approximate backward computation, which may be of interest to you. You will still need to add the perplexity computation (read more in part 9 about how to do this)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0Wq8hRy0UEX"
      },
      "source": [
        "import tqdm\n",
        "def repackage_hidden(h):\n",
        "    \"\"\"Wraps hidden states in new Tensors, to detach them from their history.\"\"\"\n",
        "    if isinstance(h, torch.Tensor):\n",
        "        return h.detach()\n",
        "    else:\n",
        "        return tuple(repackage_hidden(v) for v in h)\n",
        "\n",
        "def train(model, device, optimizer, train_loader, lr, epoch, log_interval):\n",
        "    model.train()\n",
        "    losses = []\n",
        "    hidden = None\n",
        "    for batch_idx, (data, label) in enumerate(tqdm.tqdm(train_loader)):\n",
        "        data, label = data.to(device), label.to(device)\n",
        "        # Separates the hidden state across batches. \n",
        "        # Otherwise the backward would try to go all the way to the beginning every time.\n",
        "        if hidden is not None:\n",
        "            hidden = repackage_hidden(hidden)\n",
        "        optimizer.zero_grad()\n",
        "        output, hidden = model(data)\n",
        "        pred = output.max(-1)[1]\n",
        "        loss = model.loss(output, label)\n",
        "        losses.append(loss.item())\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        if batch_idx % log_interval == 0:\n",
        "            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
        "                epoch, batch_idx * len(data), len(train_loader.dataset),\n",
        "                100. * batch_idx / len(train_loader), loss.item()))\n",
        "    return np.mean(losses)\n",
        "\n",
        "\n",
        "def test(model, device, test_loader):\n",
        "    model.eval()\n",
        "    test_loss = 0\n",
        "    correct = 0\n",
        "\n",
        "    with torch.no_grad():\n",
        "        hidden = None\n",
        "        for batch_idx, (data, label) in enumerate(test_loader):\n",
        "            data, label = data.to(device), label.to(device)\n",
        "            output, hidden = model(data, hidden)\n",
        "            test_loss += model.loss(output, label, reduction='mean').item()\n",
        "            pred = output.max(-1)[1]\n",
        "            correct_mask = pred.eq(label.view_as(pred))\n",
        "            num_correct = correct_mask.sum().item()\n",
        "            correct += num_correct\n",
        "            # Comment this out to avoid printing test results\n",
        "            if batch_idx % 10 == 0:\n",
        "                print('Input\\t%s\\nGT\\t%s\\npred\\t%s\\n\\n' % (\n",
        "                    test_loader.dataset.vocab.array_to_words(data[0]),\n",
        "                    test_loader.dataset.vocab.array_to_words(label[0]),\n",
        "                    test_loader.dataset.vocab.array_to_words(pred[0])))\n",
        "\n",
        "    test_loss /= len(test_loader)\n",
        "    test_accuracy = 100. * correct / (len(test_loader.dataset) * test_loader.dataset.sequence_length)\n",
        "\n",
        "    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
        "        test_loss, correct, len(test_loader.dataset) * test_loader.dataset.sequence_length,\n",
        "        100. * correct / (len(test_loader.dataset) * test_loader.dataset.sequence_length)))\n",
        "    return test_loss, test_accuracy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "66T-Ylkg0fn1"
      },
      "source": [
        "def main():\n",
        "    SEQUENCE_LENGTH = 100\n",
        "    BATCH_SIZE = 256\n",
        "    FEATURE_SIZE = 512\n",
        "    TEST_BATCH_SIZE = 256\n",
        "    EPOCHS = 20\n",
        "    LEARNING_RATE = 0.002\n",
        "    WEIGHT_DECAY = 0.0005\n",
        "    USE_CUDA = True\n",
        "    PRINT_INTERVAL = 10\n",
        "    LOG_PATH = DATA_PATH + 'logs/log.pkl'\n",
        "\n",
        "\n",
        "    data_train = HarryPotterDataset(DATA_PATH + 'harry_potter_chars_train.pkl', SEQUENCE_LENGTH, BATCH_SIZE)\n",
        "    data_test = HarryPotterDataset(DATA_PATH + 'harry_potter_chars_test.pkl', SEQUENCE_LENGTH, TEST_BATCH_SIZE)\n",
        "    vocab = data_train.vocab\n",
        "\n",
        "    use_cuda = USE_CUDA and torch.cuda.is_available()\n",
        "\n",
        "    device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n",
        "    print('Using device', device)\n",
        "    import multiprocessing\n",
        "    num_workers = multiprocessing.cpu_count()\n",
        "    print('num workers:', num_workers)\n",
        "\n",
        "    kwargs = {'num_workers': num_workers,\n",
        "              'pin_memory': True} if use_cuda else {}\n",
        "\n",
        "    train_loader = torch.utils.data.DataLoader(data_train, batch_size=BATCH_SIZE,\n",
        "                                               shuffle=False, **kwargs)\n",
        "    test_loader = torch.utils.data.DataLoader(data_test, batch_size=TEST_BATCH_SIZE,\n",
        "                                              shuffle=False, **kwargs)\n",
        "\n",
        "    model = HarryPotterNet(data_train.vocab_size(), FEATURE_SIZE).to(device)\n",
        "\n",
        "    # Adam is an optimizer like SGD but a bit fancier. It tends to work faster and better than SGD.\n",
        "    # We will talk more about different optimization methods in class.\n",
        "    optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY)\n",
        "    start_epoch = model.load_last_model(DATA_PATH + 'checkpoints')\n",
        "\n",
        "    train_losses, test_losses, test_accuracies = pt_util.read_log(LOG_PATH, ([], [], []))\n",
        "    test_loss, test_accuracy = test(model, device, test_loader)\n",
        "\n",
        "    test_losses.append((start_epoch, test_loss))\n",
        "    test_accuracies.append((start_epoch, test_accuracy))\n",
        "\n",
        "    try:\n",
        "        for epoch in range(start_epoch, EPOCHS + 1):\n",
        "            lr = LEARNING_RATE * np.power(0.25, (int(epoch / 6)))\n",
        "            train_loss = train(model, device, optimizer, train_loader, lr, epoch, PRINT_INTERVAL)\n",
        "            test_loss, test_accuracy = test(model, device, test_loader)\n",
        "            train_losses.append((epoch, train_loss))\n",
        "            test_losses.append((epoch, test_loss))\n",
        "            test_accuracies.append((epoch, test_accuracy))\n",
        "            pt_util.write_log(LOG_PATH, (train_losses, test_losses, test_accuracies))\n",
        "            model.save_best_model(test_accuracy, DATA_PATH + 'checkpoints/%03d.pt' % epoch)\n",
        "            seed_words = 'Harry Potter, Voldemort, and Dumbledore walk into a bar. '\n",
        "            generated_sentence = generate_language(model, device, seed_words, 200, vocab, 'max')\n",
        "            print('generated max\\t\\t', generated_sentence)\n",
        "            for ii in range(10):\n",
        "                generated_sentence = generate_language(model, device, seed_words, 200, vocab, 'sample')\n",
        "                print('generated sample\\t', generated_sentence)\n",
        "            generated_sentence = generate_language(model, device, seed_words, 200, vocab, 'beam')\n",
        "            print('generated beam\\t\\t', generated_sentence)\n",
        "            print('')\n",
        "\n",
        "    except KeyboardInterrupt as ke:\n",
        "        print('Interrupted')\n",
        "    except:\n",
        "        import traceback\n",
        "        traceback.print_exc()\n",
        "    finally:\n",
        "        print('Saving final model')\n",
        "        model.save_model(DATA_PATH + 'checkpoints/%03d.pt' % epoch, 0)\n",
        "        ep, val = zip(*train_losses)\n",
        "        pt_util.plot(ep, val, 'Train loss', 'Epoch', 'Error')\n",
        "        ep, val = zip(*test_losses)\n",
        "        pt_util.plot(ep, val, 'Test loss', 'Epoch', 'Error')\n",
        "        ep, val = zip(*test_accuracies)\n",
        "        pt_util.plot(ep, val, 'Test accuracy', 'Epoch', 'Error')\n",
        "        return model, vocab, device\n",
        "\n",
        "final_model, vocab, device = main()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r597GUTVjwZc"
      },
      "source": [
        "#Part 7: Experiments"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zgLylYlp9kBK"
      },
      "source": [
        "def eval_final_model(model, vocab, device):\n",
        "    seed_words = 'Harry Potter and the'\n",
        "    sequence_length = 200\n",
        "\n",
        "    generated_sentence = generate_language(model, device, seed_words, sequence_length, vocab, 'max')\n",
        "    print('generated with max\\t', generated_sentence)\n",
        "\n",
        "    for ii in range(10):\n",
        "        generated_sentence = generate_language(model, device, seed_words, sequence_length, vocab, 'sample')\n",
        "        print('generated with sample\\t', generated_sentence)\n",
        "\n",
        "    for ii in range(10):\n",
        "        generated_sentence = generate_language(model, device, seed_words, sequence_length, vocab, 'beam')\n",
        "        print('generated with beam\\t', generated_sentence)\n",
        "\n",
        "eval_final_model(final_model, vocab, device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pn0RWPBFjzkP"
      },
      "source": [
        "#Part 8: Other things\n",
        "Choose **three** of the following to try. You should create new code cells below rather than modifying your earlier code:\n",
        "\n",
        "\n",
        "1. Train on a different text corpus. The corpus should be at least as large as the provided Harry Potter dataset.\n",
        "    - Options include other books, websites, tweets, wikipedia articles etc.\n",
        "    -  (Hint: this is probably the easiest one)\n",
        "1. Find a better network architecture. Some ideas:\n",
        "    - Look at the various options in https://pytorch.org/docs/stable/nn.html#torch.nn.GRU\n",
        "    - You can make the network wider or deeper.\n",
        "    - You can try to learn separate encoders and decoders.\n",
        "1. Use an LSTM instead of a GRU.\n",
        "    - https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM\n",
        "    - The output of an LSTM is a hidden state **and** a cell state, so you will need to deal with a tuple instead of a single vector.\n",
        "1. Use a Transformer instead of a GRU.\n",
        "    - https://pytorch.org/docs/stable/nn.html#torch.nn.Transformer\n",
        "    - Be sure to get the masks right so you don't condition on the future characters: https://pytorch.org/docs/stable/nn.html#torch.nn.Transformer.generate_square_subsequent_mask may be of use.\n",
        "1. Train a model using student-forcing.\n",
        "    - You will have to modify the network inputs.\n",
        "    - You will need to use `torch.nn.GRUCell` and its like. https://pytorch.org/docs/stable/nn.html#grucell\n",
        "    - You cannot simply feed an empty string to start off a sequence. The sequence must be somehow conditioned on prior ground truth.\n",
        "1. Train a model on words instead of characters.\n",
        "    - You will need to redefine your input/output space vocabulary as well.\n",
        "    - You should replace any words that occur less than 5 times in the dataset with an <unknown\\> token. \n",
        "1. Write a new data loader which picks a random point in the text to start from and returns 10 consecutive sequences starting from that point onward. \n",
        "    - You should also modify the train and test functions to reset the memory when you reset the sequence.\n",
        "    - You should consider an epoch to be feeding in approximately the number of characters in the dataset.\n",
        "    - You may run into issues if your dataset size/epochs are not a multiple of your batch size.\n",
        "1. Train on sentences instead of one long sequence.\n",
        "    - You should still produce output character by character.\n",
        "    - Sentences can end with a . ! ?, but words like Mr. generally do not end a sentence.\n",
        "    - A sentence may also continue in the case of quotations. For example: ``\"Do your homework!\" said the TAs.`` is only one sentence.\n",
        "    - Your parsing does not have to be perfect, but try to incorporate as many of these rules as you can.\n",
        "    - Feel free to use existing NLP tools for finding sentence endings. One is spacy: https://spacy.io/usage/linguistic-features#section-sbd\n",
        "    - All sentences should end with an <eos\\> token. Your output sampling should now stop when it produces the <eos\\> token.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vWMlB2U3onZ0"
      },
      "source": [
        "#Part 9: Short answer questions\n",
        "Please answer these questions, and put the answers in a file called short_answer.pdf in your repository.\n",
        "\n",
        "\n",
        "1. Just like last time, provide plots for training error, test error, and test accuracy. Also provide a plot of your train and test perplexity per epoch.\n",
        "    - In class we defined perplexity as `2^(p*log_2(q))`, However the PyTorch cross entropy function uses the natural log. To compute perplexity directly from the cross entropy, you should use `e^p*ln(q)`.\n",
        "    - We encourage you to try multiple network modifications and hyperparameters, but you only need to provide plots for your best model. Please list the modifications and hyperparameters.    \n",
        "    \n",
        "2. What was your final test accuracy? What was your final test perplexity?\n",
        "3. What was your favorite sentence generated via each of the sampling methods? What was the prompt you gave to generate that sentence?\n",
        "4. Which sampling method seemed to generate the best results? Why do you think that is?\n",
        "5. For sampling and beam search, try multiple temperatures between 0 and 2. \n",
        "    - Which produces the best outputs? Best as in made the most sense, your favorite, or funniest, doesn't really matter how you decide.\n",
        "    - What does a temperature of 0 do? What does a temperature of 0<temp<1 do? What does a temperature of 1 do? What does a temperature of above 1 do? What would a negative temperature do (assuming the code allowed for negative temperature)?\n",
        "    \n",
        "Questions for each of the \"Other things\" sections. Only answer the questions corresponding to the ones you chose.\n",
        "\n",
        "1. New Corpus\n",
        "    1. What corpus did you choose? How many characters were in it?\n",
        "    2. What differences did you notice between the sentences generated with the new/vs old corpus.\n",
        "    3. Provide outputs for each sampling method on the new corpus (you can pick one temperature, but say what it was).\n",
        "\n",
        "1. New Architecture\n",
        "    1. What was your design? What did you try that didn't work well?\n",
        "    2. What was your lowest test perplexity? Provide training and testing plots.\n",
        "    3. Provide outputs for each sampling method on the new corpus (you can pick one temperature, but say what it was).\n",
        "\n",
        "1. LSTM\n",
        "    1. What new difficulties did you run into while training?\n",
        "    2. Were results better than the GRU? Provide training and testing plots.\n",
        "    3. Provide outputs for each sampling method on the new corpus (you can pick one temperature, but say what it was).\n",
        "\n",
        "1. Transformer\n",
        "    1. What new difficulties did you run into while training?\n",
        "    2. Were results better than the GRU? Provide training and testing plots.\n",
        "    3. Provide outputs for each sampling method on the new corpus (you can pick one temperature, but say what it was).\n",
        "    \n",
        "1. Student-forcing\n",
        "    1. What new difficulties did you run into while training?\n",
        "    2. Were the results better than with teacher-forcing?\n",
        "    3. Provide some outputs for each sampling method (you can pick one temperature, but say what it was).\n",
        "    \n",
        "1. Words\n",
        "    1. What new difficulties did you run into while training?\n",
        "    2. How large was your vocabulary?\n",
        "    3. Did you find that different batch size, sequence length, and feature size and other hyperparameters were needed? If so, what worked best for you?\n",
        "\n",
        "1. Random Dataloader\n",
        "    1. What new difficulties did you run into while training?\n",
        "    2. Were the results better than with the original dataloader?\n",
        "    3. Provide some outputs for each sampling method (you can pick one temperature, but say what it was). \n",
        "    \n",
        "1. Sentences\n",
        "    1. What new difficulties did you run into while training? What new difficulties did you run into while preprocessing?\n",
        "    2. Were the results better than with the original dataloader?\n",
        "    3. Provide some outputs for each sampling method (you can pick one temperature, but say what it was). \n",
        "\n",
        "\n",
        "    "
      ]
    }
  ]
}